Repository navigation
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request refines the CRC32C checksum offloading logic in the asyncio multi-range downloader. Instead of offloading all large chunks, it now defaults to offloading only when chunks reach their destination as a memoryview, with an override available via the GCS_PYTHON_CLIENT_CRC32C_OFFLOAD environment variable. The review feedback suggests optimizing the unit tests by caching patchers in _create_view_response to prevent nested patching and redundant cleanup registrations when the helper is called multiple times in a single test.
…py reads Read chunks of 1 MiB or more are hashed on the CRC32C worker pool by default only when they reach a write() destination as a memoryview: the zero-copy parser with a google-crc32c that accepts memoryview, and accepts_memoryview=True. Every other chunk is verified inline on the event-loop thread again, so callers that have not opted in to zero-copy reads keep verifying inline. Buffer destinations copy each chunk on the loop thread, so they stay inline by default too. GCS_PYTHON_CLIENT_CRC32C_OFFLOAD overrides the default: 1 (or true, yes, on) offloads every chunk of 1 MiB or more outside whole-object ranges, and any other non-empty value none. The variable is read when a chunk arrives, so a library can set it before it opens a stream. The threshold is fixed at google-crc32c's GIL-release threshold (1 MiB), and GCS_PYTHON_CLIENT_CRC32C_OFFLOAD_MIN_BYTES is removed. In 5-round SDK and gcsfs read microbenchmarks, offloading memoryview chunks gained a median 35% on SDK reads with 4-16 streams per event loop and 30% on gcsfs cat_ranges. When each chunk was still copied on the loop thread, results were mixed: SDK reads with 4-16 streams per loop still gained 10% with the generated parser, but gcsfs reads of 1 MiB or more lost in 10 of 18 cases, and single-threaded gcsfs reads lost a median 7% with accepts_memoryview=False.
yuxin00j
force-pushed
the
storage-offloop-opt-in
branch
from
November 16, 2026 12:39
6f87228 to
9147a64
Compare
Contributor
Author
|
/gcbrun |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This narrows the off-loop CRC32C verification from #18628 before it ships, so it is on by default only for zero-copy reads.
write()destination asmemoryview(the zero-copy parser with a google-crc32c that acceptsmemoryview, andaccepts_memoryview=True) are verified on the worker pool, as in perf(storage): verify CRC32C of large read chunks off the event-loop thread #18628. Every other chunk is verified inline again.GCS_PYTHON_CLIENT_CRC32C_OFFLOAD=1offloads every chunk of 1 MiB or more outside whole-object ranges, andGCS_PYTHON_CLIENT_CRC32C_OFFLOAD=0none.Offloading pays off once no
bytescopy of each chunk is left on the event-loop thread. With such a copy (the generated parser oraccepts_memoryview=False), results are mixed: SDK multi-stream reads and gcsfscat_rangesstill gain, but gcsfsreadof 1 MiB or more loses in 10 of 18 cases with the generated parser.Changes
GCS_PYTHON_CLIENT_CRC32C_OFFLOADreplacesGCS_PYTHON_CLIENT_CRC32C_OFFLOAD_MIN_BYTES, which never shipped.1,true,yesoronoffloads every chunk of 1 MiB or more outside whole-object ranges, and any other value none.write()destination withaccepts_memoryview=True). Buffer destinations still copy each chunk on the loop thread, so they stay inline by default.enable_checksumdocstring ofdownload_rangesdescribes the default and the override.test_mrd_zero_copy_bidi_readsets the flag to1, so the Rapid system test covers the offload path even with a google-crc32c that rejectsmemoryview.Testing
cd packages/google-cloud-storage ruff check --select I --target-version=py310 --line-length=88 docs google tests noxfile.py setup.py ruff format --check --target-version=py310 --line-length=88 docs google tests noxfile.py setup.py flake8 google tests pytest tests/unit tests/resumable_media/unit PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python pytest tests/unit/asyncio